{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## インストール"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### dockerのインストール\n",
    "ansibleで構築。いずれ追記予定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### swarm manager設定と起動\n",
    "\n",
    "```\n",
    "docker swarm init --advertise-addr 10.136.3.183\n",
    "================================================\n",
    "Swarm initialized: current node (r0cr3pb7xllmom19lhx02mu0e) is now a manager.\n",
    "\n",
    "To add a worker to this swarm, run the following command:\n",
    "\n",
    "    docker swarm join --token SWMTKN-1-5ya5hny9r7c3lfq57a4bjaamb78maeo17jicq8s70ik94ptzfs-eiz0nl7i1ilvxy35ttwypvwii 10.136.3.183:2377\n",
    "\n",
    "To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.\n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### swarmクラスタにノードを追加\n",
    "\n",
    "```\n",
    "### 各ノードで以下実行\n",
    "$ docker swarm join --token SWMTKN-1-5ya5hny9r7c3lfq57a4bjaamb78maeo17jicq8s70ik94ptzfs-eiz0nl7i1ilvxy35ttwypvwii 10.136.3.183:2377\n",
    "================================================\n",
    "This node joined a swarm as a worker.\n",
    "================================================\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 追加されたノードの確認\n",
    "\n",
    "swarm managerのノードで実行\n",
    "\n",
    "```\n",
    "### ノードの一覧表示\n",
    "$ docker node ls\n",
    "================================================\n",
    "ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS\n",
    "aq3jx7y5s8h7rylc2c086woih     swarm-node2    Ready               Active\n",
    "fjuw9s9xgzyt3d69byydcmrb3     swarm-node1    Ready               Active\n",
    "r0cr3pb7xllmom19lhx02mu0e *   swarm-master   Ready               Active              Leader\n",
    "================================================\n",
    "\n",
    "### swarmの状態の確認\n",
    "$ docker info\n",
    "================================================\n",
    "Containers: 1\n",
    " Running: 0\n",
    " Paused: 0\n",
    " Stopped: 1\n",
    "Images: 1\n",
    "Server Version: 17.06.0-ce\n",
    "Storage Driver: overlay\n",
    " Backing Filesystem: xfs\n",
    " Supports d_type: false\n",
    "Logging Driver: json-file\n",
    "Cgroup Driver: cgroupfs\n",
    "Plugins:\n",
    " Volume: local\n",
    " Network: bridge host macvlan null overlay\n",
    " Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\n",
    "Swarm: active\n",
    " NodeID: r0cr3pb7xllmom19lhx02mu0e\n",
    " Is Manager: true\n",
    " ClusterID: vbiiozezbop3u65rpd6ycffi1\n",
    " Managers: 1\n",
    " Nodes: 3\n",
    " Orchestration:\n",
    "  Task History Retention Limit: 5\n",
    " Raft:\n",
    "  Snapshot Interval: 10000\n",
    "  Number of Old Snapshots to Retain: 0\n",
    "  Heartbeat Tick: 1\n",
    "  Election Tick: 3\n",
    " Dispatcher:\n",
    "  Heartbeat Period: 5 seconds\n",
    " CA Configuration:\n",
    "  Expiry Duration: 3 months\n",
    "  Force Rotate: 0\n",
    " Root Rotation In Progress: false\n",
    " Node Address: 10.136.3.183\n",
    " Manager Addresses:\n",
    "  10.136.3.183:2377\n",
    "Runtimes: runc\n",
    "Default Runtime: runc\n",
    "Init Binary: docker-init\n",
    "containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a\n",
    "runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4\n",
    "init version: 949e6fa\n",
    "Security Options:\n",
    " seccomp\n",
    "  Profile: default\n",
    "Kernel Version: 3.10.0-514.26.2.el7.x86_64\n",
    "Operating System: CentOS Linux 7 (Core)\n",
    "OSType: linux\n",
    "Architecture: x86_64\n",
    "CPUs: 4\n",
    "Total Memory: 3.859GiB\n",
    "Name: swarm-master\n",
    "ID: 2S7J:TN46:GS5K:O4PF:PPDE:N3HT:FDWE:DCMX:AG7F:4NV6:H6AI:4RSM\n",
    "Docker Root Dir: /var/lib/docker\n",
    "Debug Mode (client): false\n",
    "Debug Mode (server): false\n",
    "Registry: https://index.docker.io/v1/\n",
    "Experimental: false\n",
    "Insecure Registries:\n",
    " 127.0.0.0/8\n",
    "Live Restore Enabled: false\n",
    "\n",
    "WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.\n",
    "         Reformat the filesystem with ftype=1 to enable d_type support.\n",
    "         Running without d_type support will not be supported in future releases.\n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## サービス作成\n",
    "\n",
    "swarm managerのホストで実行\n",
    "\n",
    "* オプション\n",
    "    * docker service create\n",
    "        * サービスを作成する\n",
    "    * --replicas 1\n",
    "        * 少なくとも1つのインスタンスが動いていることを希望する\n",
    "    * --name\n",
    "        * サービス名\n",
    "    * alpine ping docker.com\n",
    "        * Alpine Linuxでdocker.comにpingを実行するようにサービスが定義されている\n",
    "\n",
    "```\n",
    "### サービスを作成する\n",
    "$ docker service create --replicas 1 --name helloworld alpine ping docker.com\n",
    "================================================\n",
    "jknp98np9ujtt9hn5rs7rizch\n",
    "Since --detach=false was not specified, tasks will be created in the background.\n",
    "In a future release, --detach=false will become the default.\n",
    "================================================\n",
    "\n",
    "### サービスの確認\n",
    "# docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "jknp98np9ujt        helloworld          replicated          1/1                 alpine:latest\n",
    "================================================\n",
    "\n",
    "### サービスの詳細を確認\n",
    "### 「--pretty」がなければjsonフォーマットになる\n",
    "# docker service inspect --pretty helloworld\n",
    "================================================\n",
    "\n",
    "ID:\t\tjknp98np9ujtt9hn5rs7rizch\n",
    "Name:\t\thelloworld\n",
    "Service Mode:\tReplicated\n",
    " Replicas:\t1\n",
    "Placement:\n",
    "UpdateConfig:\n",
    " Parallelism:\t1\n",
    " On failure:\tpause\n",
    " Monitoring Period: 5s\n",
    " Max failure ratio: 0\n",
    " Update order:      stop-first\n",
    "RollbackConfig:\n",
    " Parallelism:\t1\n",
    " On failure:\tpause\n",
    " Monitoring Period: 5s\n",
    " Max failure ratio: 0\n",
    " Rollback order:    stop-first\n",
    "ContainerSpec:\n",
    " Image:\t\talpine:latest@sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe\n",
    " Args:\t\tping docker.com\n",
    "Resources:\n",
    "Endpoint Mode:\tvip\n",
    "================================================\n",
    "\n",
    "### どのノードでサービスが動いているのか確認する\n",
    "$ docker service ps helloworld\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS\n",
    "2a2lmia49vct        helloworld.1        alpine:latest       swarm-master   Running             Running 14 minutes ago\n",
    "================================================\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### サービスのスケール変更\n",
    "\n",
    "```\n",
    "### hello worldを5つのインスタンスで実行する状態にする\n",
    "$ docker service scale helloworld=5\n",
    "================================================\n",
    "helloworld scaled to 5\n",
    "================================================\n",
    "\n",
    "### 起動しているホストとサービスの確認\n",
    "$ docker service ps helloworld\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS\n",
    "2a2lmia49vct        helloworld.1        alpine:latest       swarm-master   Running             Running 21 minutes ago\n",
    "hbmzxlmpnhde        helloworld.2        alpine:latest       swarm-node2    Running             Running about a minute ago\n",
    "tqtyi6evlybo        helloworld.3        alpine:latest       swarm-node1    Running             Running about a minute ago\n",
    "uc1am1kkbqr8        helloworld.4        alpine:latest       swarm-node1    Running             Running about a minute ago\n",
    "x7p58j9krnqu        helloworld.5        alpine:latest       swarm-master   Running             Running about a minute ago\n",
    "================================================\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## サービスの削除\n",
    "\n",
    "```\n",
    "### サービスの削除\n",
    "$ docker service rm helloworld\n",
    "================================================\n",
    "helloworld\n",
    "================================================\n",
    "\n",
    "### サービスが無いことの確認\n",
    "$ docker service ps helloworld\n",
    "================================================\n",
    "no such services: helloworld\n",
    "================================================\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## サービスのローリングアップデート\n",
    "\n",
    "Redisを3.0.6から3.0.7にアップデートする\n",
    "\n",
    "* オプション\n",
    "    * --update-delay 10s\n",
    "        * updateするときに10秒ずらして実施する\n",
    "    * --update-parallelism 3\n",
    "        * 最大で同時に3つのサービスをupdateできる\n",
    "        * デフォルトは1サービスずつ\n",
    "    * --update-failure-action\n",
    "        * updateに失敗した時のアクションを指定できる\n",
    "        \n",
    "\n",
    "```\n",
    "### Redis3.0.6のサービスを起動する\n",
    "$ docker service create --replicas 3 --name redis --update-delay 10s redis:3.0.6\n",
    "================================================\n",
    "bxaeq7cs1pcrufi90z368qauq\n",
    "Since --detach=false was not specified, tasks will be created in the background.\n",
    "In a future release, --detach=false will become the default.\n",
    "================================================\n",
    "\n",
    "### 状況の確認\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS\n",
    "2x9ns0cb26jj        redis.1             redis:3.0.6         kubernetes-node2    Running             Preparing 8 seconds ago\n",
    "gfa0m5wov5lv        redis.2             redis:3.0.6         kubernetes-master   Running             Preparing 8 seconds ago\n",
    "sqzq2jl9skms        redis.3             redis:3.0.6         kubernetes-node1    Running             Preparing 8 seconds ago\n",
    "================================================\n",
    "\n",
    "### ローリングアップデートを開始する\n",
    "$ docker service update --image redis:3.0.7 redis\n",
    "================================================\n",
    "redis\n",
    "Since --detach=false was not specified, tasks will be updated in the background.\n",
    "In a future release, --detach=false will become the default.\n",
    "================================================\n",
    "\n",
    "### 状況の確認\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS\n",
    "8wvx4psiowpi        redis.1             redis:3.0.6         kubernetes-master   Running             Running 25 minutes ago\n",
    "kquj43szvt2j        redis.2             redis:3.0.7         kubernetes-node1    Running             Preparing 7 seconds ago\n",
    "xdvc8k4zloxh         \\_ redis.2         redis:3.0.6         kubernetes-node1    Shutdown            Shutdown 6 seconds ago\n",
    "e8t4dlqh0dqh        redis.3             redis:3.0.6         kubernetes-node2    Running             Running 25 minutes ago\n",
    "================================================\n",
    "\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                 ERROR               PORTS\n",
    "9s1v8j1a97es        redis.1             redis:3.0.7         kubernetes-master   Running             Running about a minute ago\n",
    "8wvx4psiowpi         \\_ redis.1         redis:3.0.6         kubernetes-master   Shutdown            Shutdown about a minute ago\n",
    "kquj43szvt2j        redis.2             redis:3.0.7         kubernetes-node1    Running             Running 2 minutes ago\n",
    "xdvc8k4zloxh         \\_ redis.2         redis:3.0.6         kubernetes-node1    Shutdown            Shutdown 4 minutes ago\n",
    "nrwnain18s80        redis.3             redis:3.0.7         kubernetes-node2    Running             Running 26 seconds ago\n",
    "e8t4dlqh0dqh         \\_ redis.3         redis:3.0.6         kubernetes-node2    Shutdown            Shutdown 57 seconds ago\n",
    "================================================\n",
    "\n",
    "### サービスの詳細確認\n",
    "$ docker service inspect --pretty redis\n",
    "================================================\n",
    "ID:\t\tbxaeq7cs1pcrufi90z368qauq\n",
    "Name:\t\tredis\n",
    "Service Mode:\tReplicated\n",
    " Replicas:\t3\n",
    "UpdateStatus:\n",
    " State:\t\tcompleted\n",
    " Started:\t7 minutes ago\n",
    " Completed:\t3 minutes ago\n",
    " Message:\tupdate completed\n",
    "Placement:\n",
    "UpdateConfig:\n",
    " Parallelism:\t1\n",
    " Delay:\t\t10s\n",
    " On failure:\tpause\n",
    " Monitoring Period: 5s\n",
    " Max failure ratio: 0\n",
    " Update order:      stop-first\n",
    "RollbackConfig:\n",
    " Parallelism:\t1\n",
    " On failure:\tpause\n",
    " Monitoring Period: 5s\n",
    " Max failure ratio: 0\n",
    " Rollback order:    stop-first\n",
    "ContainerSpec:\n",
    " Image:\t\tredis:3.0.7@sha256:730b765df9fe96af414da64a2b67f3a5f70b8fd13a31e5096fee4807ed802e20\n",
    "Resources:\n",
    "Endpoint Mode:\tvip\n",
    "================================================\n",
    "\n",
    "$ docker ps -a\n",
    "================================================\n",
    "CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES\n",
    "6864b1f9c1a6        redis:3.0.7         \"docker-entrypoint...\"   13 minutes ago      Up 13 minutes               6379/tcp            redis.1.9s1v8j1a97esd6cimkyouf6gv\n",
    "abbc83ee4cd7        redis:3.0.6         \"/entrypoint.sh re...\"   42 minutes ago      Exited (0) 14 minutes ago                       redis.1.8wvx4psiowpigc69vp14rkg9q\n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 特定のノードのコンテナを他に移す\n",
    "\n",
    "* メンテナンスなど\n",
    "* ここではswarm-node1のコンテナを他に移す\n",
    "\n",
    "```\n",
    "### 現状の確認\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS\n",
    "dyhxen4d5yk3        redis.1             redis:3.0.6         swarm-node2    Running             Running 7 seconds ago\n",
    "sncpngpito8z        redis.2             redis:3.0.6         swarm-master   Running             Running 7 seconds ago\n",
    "zdj8hlf8q958        redis.3             redis:3.0.6         swarm-node1    Running             Running 7 seconds ago\n",
    "================================================\n",
    "\n",
    "### swarm-node1のコンテナを他に移す\n",
    "$ docker node update --availability drain swarm-node1\n",
    "================================================\n",
    "swarm-node1\n",
    "================================================\n",
    "\n",
    "### 状況の確認\n",
    "$ docker node ls\n",
    "================================================\n",
    "ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS\n",
    "v9yx1vm5apm75j54j54dc36as     swarm-node2    Ready               Active\n",
    "wj75qtvh5m644bl7bn6jfmcru *   swarm-master   Ready               Active              Leader\n",
    "wxi6n7fhgvctraalihfinq8pf     swarm-node1    Ready               Drain\n",
    "================================================\n",
    "\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS\n",
    "dyhxen4d5yk3        redis.1             redis:3.0.6         swarm-node2    Running             Running 44 minutes ago\n",
    "sncpngpito8z        redis.2             redis:3.0.6         swarm-master   Running             Running 44 minutes ago\n",
    "88os58fhkh1o        redis.3             redis:3.0.6         swarm-node2    Running             Running 6 seconds ago\n",
    "zdj8hlf8q958         \\_ redis.3         redis:3.0.6         swarm-node1    Shutdown            Shutdown 6 seconds ago\n",
    "================================================\n",
    "\n",
    "$ docker node inspect --pretty swarm-node1\n",
    "================================================\n",
    "ID:                     wxi6n7fhgvctraalihfinq8pf\n",
    "Hostname:               swarm-node1\n",
    "Joined at:              2017-07-24 00:22:05.341823651 +0000 utc\n",
    "Status:\n",
    " State:                 Ready\n",
    " Availability:          Drain\n",
    " Address:               10.136.3.184\n",
    "Platform:\n",
    " Operating System:      linux\n",
    " Architecture:          x86_64\n",
    "Resources:\n",
    " CPUs:                  4\n",
    " Memory:                7.797GiB\n",
    "Plugins:\n",
    " Log:           awslogs, fluentd, gcplogs, gelf, journald, json-file, logentries, splunk, syslog\n",
    " Network:               bridge, host, macvlan, null, overlay\n",
    " Volume:                local\n",
    "Engine Version:         17.06.0-ce\n",
    "TLS Info:\n",
    " TrustRoot:\n",
    "-----BEGIN CERTIFICATE-----\n",
    "MIIBaTCCARCgAwIBAgIUE0KlQJ+QSAiGTPQCZ7dwNBYDaJYwCgYIKoZIzj0EAwIw\n",
    "...\n",
    "ms4Cy/7mWMZmN6HQ4alAqiMvdUxR8sVTZQhjz30=\n",
    "-----END CERTIFICATE-----\n",
    "\n",
    " Issuer Subject:        MBMxETAPBgNVBAMTCHN3YXJtLWNh\n",
    " Issuer Public Key:     MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE+LttLn/Nt...d6duAsdD541WLEcWjjqNPBtRgB7EMNZ28/jD9WA==\n",
    "================================================\n",
    "\n",
    "### ノードをActiveに戻す\n",
    "$ docker node update --availability active swarm-node1\n",
    "================================================\n",
    "swarm-node1\n",
    "================================================\n",
    "\n",
    "### 状況の確認\n",
    "$ docker node ls\n",
    "================================================\n",
    "ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS\n",
    "v9yx1vm5apm75j54j54dc36as     swarm-node2    Ready               Active\n",
    "wj75qtvh5m644bl7bn6jfmcru *   swarm-master   Ready               Active              Leader\n",
    "wxi6n7fhgvctraalihfinq8pf     swarm-node1    Ready               Active\n",
    "================================================\n",
    "\n",
    "### コンテナはそのまま(元のホストに戻るわけでは無い)\n",
    "$ docker service ps redis\n",
    "================================================\n",
    "ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS\n",
    "dyhxen4d5yk3        redis.1             redis:3.0.6         swarm-node2    Running             Running 44 minutes ago\n",
    "sncpngpito8z        redis.2             redis:3.0.6         swarm-master   Running             Running 44 minutes ago\n",
    "88os58fhkh1o        redis.3             redis:3.0.6         swarm-node2    Running             Running 6 seconds ago\n",
    "zdj8hlf8q958         \\_ redis.3         redis:3.0.6         swarm-node1    Shutdown            Shutdown 6 seconds ago\n",
    "================================================\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ロードバランサの設定\n",
    "\n",
    "nginxのコンテナを2付きどうしてそれぞれにアクセスするようにする\n",
    "\n",
    "```\n",
    "### サービスのportを公開する\n",
    "$ docker service create --name my-web --publish 8080:80 --replicas 2 nginx\n",
    "$ docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "zv7i78yyd6tz        my-web              replicated          2/2                 nginx:latest        *:8080->80/tcp\n",
    "================================================\n",
    "\n",
    "### 公開しているportを追加する\n",
    "$ docker service update --publish-add 8081:80 my-web\n",
    "$ docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "zv7i78yyd6tz        my-web              replicated          2/2                 nginx:latest        *:8080->80/tcp,*:8081->80/tcp\n",
    "================================================\n",
    "\n",
    "### 詳細情報の確認\n",
    "$ docker service inspect --format=\"{{json .Endpoint.Spec.Ports}}\"  my-web | jq\n",
    "================================================\n",
    "[\n",
    "  {\n",
    "    \"Protocol\": \"tcp\",\n",
    "    \"TargetPort\": 80,\n",
    "    \"PublishedPort\": 8080,\n",
    "    \"PublishMode\": \"ingress\"\n",
    "  },\n",
    "  {\n",
    "    \"Protocol\": \"tcp\",\n",
    "    \"TargetPort\": 80,\n",
    "    \"PublishedPort\": 8081,\n",
    "    \"PublishMode\": \"ingress\"\n",
    "  }\n",
    "]\n",
    "================================================\n",
    "\n",
    "### 公開しているportを削除する\n",
    "### portはコンテナ側を指定する\n",
    "$ docker service update --publish-rm 80 my-web\n",
    "$ docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "zv7i78yyd6tz        my-web              replicated          2/2                 nginx:latest        \n",
    "================================================\n",
    "※ 「8081:80」で指定しても「8080:80」も一緒に消える\n",
    "\n",
    "### udpの指定 (デフォルトはtcpのみ。udpは明示的に指定する必要がある)\n",
    "$ docker service update --publish-add 8081:80/udp my-web\n",
    "$ docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "zv7i78yyd6tz        my-web              replicated          2/2                 nginx:latest        *:8081->80/udp\n",
    "================================================\n",
    "$ docker service update --publish-rm 80/udp my-web\n",
    "$ docker service ls\n",
    "================================================\n",
    "ID                  NAME                MODE                REPLICAS            IMAGE               PORTS\n",
    "zv7i78yyd6tz        my-web              replicated          2/2                 nginx:latest        \n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## swarmの管理\n",
    "\n",
    "swarm managerのホストで実行\n",
    "\n",
    "```\n",
    "### tokenの表示\n",
    "$ docker swarm join-token worker\n",
    "================================================\n",
    "To add a worker to this swarm, run the following command:\n",
    "\n",
    "    docker swarm join --token SWMTKN-1-5ya5hny9r7c3lfq57a4bjaamb78maeo17jicq8s70ik94ptzfs-eiz0nl7i1ilvxy35ttwypvwii 10.136.3.183:2377\n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (以下試験的な手順)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goのインストール\n",
    "\n",
    "* バージョンは1.7以上\n",
    "    * CentOS7はyumで入れると1.6が入ってしまうので使えない\n",
    "\n",
    "```\n",
    "$ wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz\n",
    "$ tar zxvf go1.8.linux-amd64.tar.gz\n",
    "$ sudo chown root:root -R go\n",
    "$ sudo mv go /usr/local/\n",
    "$ sudo vi /etc/profile.d/go.sh\n",
    "================================================\n",
    "export GOROOT=/usr/local/go\n",
    "export PATH=$PATH:$GOROOT/bin\n",
    "================================================\n",
    "$ source /etc/profile.d/go.sh\n",
    "\n",
    "$ mkdir ~/gocode\n",
    "$ vi .bashrc\n",
    "================================================\n",
    "export GOPATH=~/gocode\n",
    "export PATH=$PATH:$GOPATH/bin\n",
    "================================================\n",
    "$ source ~/.bashrc\n",
    "\n",
    "$ which go\n",
    "================================================\n",
    "/usr/local/go/bin/go\n",
    "================================================\n",
    "\n",
    "$ go version\n",
    "================================================\n",
    "go version go1.8 linux/amd64\n",
    "================================================\n",
    "```\n",
    "\n",
    "### 必要なライブラリのインストール\n",
    "\n",
    "```\n",
    "### vndrのインストール\n",
    "$ go get github.com/LK4d4/vndr\n",
    "\n",
    "### golintのインストール\n",
    "$ go get github.com/golang/lint/golint\n",
    "```\n",
    "\n",
    "### Swarmリポジトリをcloneする\n",
    "\n",
    "```\n",
    "### swarmをclone\n",
    "$ mkdir -p $GOPATH/src/github.com/docker\n",
    "$ cd $GOPATH/src/github.com/docker\n",
    "$ git clone https://github.com/docker/swarm.git\n",
    "$ cd swarm\n",
    "\n",
    "### originを自分のリポジトリに指定\n",
    "$ git remote remove origin\n",
    "$ git remote add origin https://github.com/ftakao2007/swarm.git\n",
    "$ git remote add upstream https://github.com/docker/swarm.git\n",
    "$ git remote set-url --push upstream no-pushing\n",
    "$ git remote -v\n",
    "================================================\n",
    "origin  https://github.com/ftakao2007/swarm.git (fetch)\n",
    "origin  https://github.com/ftakao2007/swarm.git (push)\n",
    "upstream        https://github.com/docker/swarm.git (fetch)\n",
    "upstream        no-pushing (push)\n",
    "================================================\n",
    "\n",
    "### 最新を自分のforkに取り込む場合\n",
    "### $ git fetch upstream master\n",
    "### $ git rebase upstream/master\n",
    "### $ git push -f origin master\n",
    "```\n",
    "\n",
    "### Swarmバイナリをビルドする\n",
    "\n",
    "```\n",
    "### ビルドする\n",
    "$ cd $GOPATH/src/github.com/docker/swarm\n",
    "$ go install .\n",
    "\n",
    "### テスト\n",
    "$ $GOPATH/bin/swarm -v\n",
    "================================================\n",
    "swarm version 1.2.8 (HEAD)\n",
    "================================================\n",
    "```\n",
    "\n",
    "### バイナリを各ホストに配布する\n",
    "\n",
    "``` \n",
    "$ scp $GOPATH/bin/swarm (対象ホスト):\n",
    "$ ssh (対象ホスト)\n",
    "$ sudo chown root:root swarm\n",
    "$ sudo mv swarm /usr/local/bin\n",
    "$ swarm -version\n",
    "================================================\n",
    "swarm version 1.2.8 (HEAD)\n",
    "================================================\n",
    "```\n",
    "\n",
    "### 参考\n",
    "\n",
    "* [公式手順](https://github.com/docker/swarm/blob/master/CONTRIBUTING.md)\n",
    "* [Docker SwarmによるDockerクラスタ環境の構築（Dockerの最新機能を使ってみよう：第4回）](http://knowledge.sakura.ad.jp/knowledge/5197/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Etcdの設定\n",
    "\n",
    "* swarm-managerのホストのみ実施\n",
    "\n",
    "```\n",
    "### Etcdのインストール\n",
    "$ sudo yum install etcd\n",
    "\n",
    "### 外部からのアクセスを受け付けるよう/etc/etcd/etcd.confを修正\n",
    "### localhostを自分のアドレスに変更\n",
    "$ cp /etc/etcd/etcd.conf /tmp\n",
    "$ sudo vi /etc/etcd/etcd.conf\n",
    "$ diff -u /tmp/etcd.conf /etc/etcd/etcd.conf\n",
    "================================================\n",
    "--- /tmp/etcd.conf      2017-07-22 11:35:46.092458672 +0900\n",
    "+++ /etc/etcd/etcd.conf 2017-07-22 11:36:27.330911397 +0900\n",
    "@@ -6,7 +6,7 @@\n",
    " #ETCD_HEARTBEAT_INTERVAL=\"100\"\n",
    " #ETCD_ELECTION_TIMEOUT=\"1000\"\n",
    " #ETCD_LISTEN_PEER_URLS=\"http://localhost:2380\"\n",
    "-ETCD_LISTEN_CLIENT_URLS=\"http://localhost:2379\"\n",
    "+ETCD_LISTEN_CLIENT_URLS=\"http://10.136.3.183:2379\"\n",
    " #ETCD_MAX_SNAPSHOTS=\"5\"\n",
    " #ETCD_MAX_WALS=\"5\"\n",
    " #ETCD_CORS=\"\"\n",
    "@@ -17,7 +17,7 @@\n",
    " #ETCD_INITIAL_CLUSTER=\"default=http://localhost:2380\"\n",
    " #ETCD_INITIAL_CLUSTER_STATE=\"new\"\n",
    " #ETCD_INITIAL_CLUSTER_TOKEN=\"etcd-cluster\"\n",
    "-ETCD_ADVERTISE_CLIENT_URLS=\"http://localhost:2379\"\n",
    "+ETCD_ADVERTISE_CLIENT_URLS=\"http://10.136.3.183:2379\"\n",
    " #ETCD_DISCOVERY=\"\"\n",
    " #ETCD_DISCOVERY_SRV=\"\"\n",
    " #ETCD_DISCOVERY_FALLBACK=\"proxy\"\n",
    "================================================\n",
    "```\n",
    "\n",
    "* 以下は全ホストで実施\n",
    "\n",
    "```\n",
    "### dockerがどの環境設定ファイルやサービス用ファイルを使っているか確認する\n",
    "### 以下の場合「/usr/lib/systemd/system/docker.service」というサービス用ファイルを使っている\n",
    "$ systemctl show docker | grep EnvironmentFile\n",
    "================================================\n",
    "(何もなし)\n",
    "================================================\n",
    "\n",
    "$ systemctl status docker | grep Loaded\n",
    "================================================\n",
    "   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)\n",
    "================================================\n",
    "\n",
    "$ grep EnvironmentFile /usr/lib/systemd/system/docker.service\n",
    "================================================\n",
    "(何もなし)\n",
    "================================================\n",
    "\n",
    "### DOCKER_NETWORK_OPTIONSを追加する\n",
    "### Serviceディレクティブに設定する\n",
    "$ cp /usr/lib/systemd/system/docker.service /tmp\n",
    "$ sudo vi /usr/lib/systemd/system/docker.service\n",
    "$ diff -u /tmp/docker.service /usr/lib/systemd/system/docker.service\n",
    "================================================\n",
    "--- /tmp/docker.service 2017-07-22 12:04:17.007548515 +0900\n",
    "+++ /usr/lib/systemd/system/docker.service      2017-07-22 12:06:11.246932774 +0900\n",
    "@@ -5,6 +5,7 @@\n",
    " Wants=network-online.target\n",
    "\n",
    " [Service]\n",
    "+EnvironmentFile=-/etc/sysconfig/docker-network\n",
    " Type=notify\n",
    " # the default is not to use systemd for cgroups because the delegate issues still\n",
    " # exists and systemd currently does not support the cgroup feature set required\n",
    "================================================\n",
    "\n",
    "### 「--cluster-store」はEtcdが動いているホストのIP、「--cluster-advertise」は各ホストのIP\n",
    "$ sudo vi /etc/sysconfig/docker-network\n",
    "================================================\n",
    "DOCKER_NETWORK_OPTIONS='--cluster-store=etcd://10.136.3.183:2379 --cluster-advertise=10.136.3.184:2376'\n",
    "================================================\n",
    "\n",
    "### デーモンをreload\n",
    "$ sudo systemctl daemon-reload\n",
    "\n",
    "### dockerを再起動\n",
    "$ sudo systemctl restart docker.service\n",
    "\n",
    "### 設定の確認\n",
    "$ systemctl show docker | grep EnvironmentFile\n",
    "================================================\n",
    "EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)\n",
    "================================================\n",
    "\n",
    "### Etcdをインストールしているホストに接続できるかcurlで確認\n",
    "$ curl -L http://10.136.3.183:2379/version\n",
    "================================================\n",
    "{\"etcdserver\":\"3.1.9\",\"etcdcluster\":\"3.1.0\"}\n",
    "================================================\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Swarm Managerの起動と接続確認"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Swarm Managerを起動する\n",
    "\n",
    "```\n",
    "### tcp://＜ホストのIPアドレス＞:3375 etcd://＜Etcdを稼働させているホストのIPアドレス＞:2379\n",
    "### Swarm ManagerのホストにEtcdをインストールすることがほとんどなので通常は同じアドレスになるはず\n",
    "$ swarm manage -H tcp://10.136.3.183:3375 etcd://10.136.3.183:2379\n",
    "================================================\n",
    "INFO[0000] Initializing discovery without TLS\n",
    "INFO[0000] Listening for HTTP                            addr=10.136.3.183:3375 proto=tcp\n",
    "================================================\n",
    "※ 検証のため手で実行しているが、運用するときはデーモン化しておく\n",
    "\n",
    "\n",
    "### 動作確認\n",
    "docker -H tcp://10.136.3.183:3375 info\n",
    "================================================\n",
    "Containers: 0\n",
    " Running: 0\n",
    " Paused: 0\n",
    " Stopped: 0\n",
    "Images: 0\n",
    "Server Version: swarm/1.2.8\n",
    "Role: primary\n",
    "Strategy: spread\n",
    "Filters: health, port, containerslots, dependency, affinity, constraint, whitelist\n",
    "Nodes: 0　　　※ まだnodeが接続されていないので0\n",
    "Plugins:\n",
    " Volume:\n",
    " Network:\n",
    "Swarm:\n",
    " NodeID:\n",
    " Is Manager: false\n",
    " Node Address:\n",
    "Kernel Version: 3.10.0-514.26.2.el7.x86_64\n",
    "Operating System: linux\n",
    "Architecture: amd64\n",
    "CPUs: 0\n",
    "Total Memory: 0B\n",
    "Name: kubernetes-master\n",
    "Docker Root Dir:\n",
    "Debug Mode (client): false\n",
    "Debug Mode (server): false\n",
    "Experimental: false\n",
    "Live Restore Enabled: false\n",
    "\n",
    "WARNING: No kernel memory limit support\n",
    "================================================\n",
    "\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 各ホスト設定\n",
    "\n",
    "```\n",
    "### OPTIONSを追加する\n",
    "### Serviceディレクティブに設定する\n",
    "$ cp /usr/lib/systemd/system/docker.service /tmp\n",
    "$ sudo vi /usr/lib/systemd/system/docker.service\n",
    "$ diff -u /tmp/docker.service /usr/lib/systemd/system/docker.service\n",
    "================================================\n",
    "--- /tmp/docker.service\t2017-07-23 07:52:25.115684930 +0900\n",
    "+++ /usr/lib/systemd/system/docker.service\t2017-07-23 07:52:48.511380621 +0900\n",
    "@@ -6,6 +6,7 @@\n",
    "\n",
    " [Service]\n",
    " EnvironmentFile=-/etc/sysconfig/docker-network\n",
    "+EnvironmentFile=-/etc/sysconfig/docker\n",
    " Type=notify\n",
    " # the default is not to use systemd for cgroups because the delegate issues still\n",
    " # exists and systemd currently does not support the cgroup feature set required\n",
    "================================================\n",
    "\n",
    "### /etc/sysconfig/dockerファイルを作成する\n",
    "### アドレスは自ホストのIPアドレス\n",
    "$ sudo vi /etc/sysconfig/docker\n",
    "================================================\n",
    "OPTIONS='-H 10.136.3.184:2375 --selinux-enabled --log-driver=journald'\n",
    "================================================\n",
    "\n",
    "### デーモンをreload\n",
    "$ sudo systemctl daemon-reload\n",
    "\n",
    "### dockerを再起動\n",
    "$ sudo systemctl restart docker.service\n",
    "\n",
    "### 設定の確認\n",
    "$ systemctl show docker | grep EnvironmentFile\n",
    "================================================\n",
    "EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)\n",
    "EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)\n",
    "================================================\n",
    "\n",
    "### 「--advertise」は自ホストのIP、「etcd」はetcdが起動しているホストのID\n",
    "swarm join --advertise=10.136.3.184:2375 etcd://10.136.3.183:2379\n",
    "================================================\n",
    "INFO[0000] Initializing discovery without TLS\n",
    "INFO[0000] Registering on the discovery service every 1m0s...  addr=10.136.3.184:2375 discovery=etcd://10.136.3.183:2379\n",
    "INFO[0060] Registering on the discovery service every 1m0s...  addr=10.136.3.184:2375 discovery=etcd://10.136.3.183:2379\n",
    "================================================\n",
    "※ 検証のため手で実行しているが、運用するときはデーモン化しておく\n",
    "\n",
    "### 接続確認\n",
    "### IPアドレスはSwarm Managerのホストを指定\n",
    "$ docker -H tcp://10.136.3.183:3375 info\n",
    "================================================\n",
    "Containers: 0\n",
    " Running: 0\n",
    " Paused: 0\n",
    " Stopped: 0\n",
    "Images: 0\n",
    "Server Version: swarm/1.2.8\n",
    "Role: primary\n",
    "Strategy: spread\n",
    "Filters: health, port, containerslots, dependency, affinity, constraint, whitelist\n",
    "Nodes: 1\n",
    " (unknown): 10.136.3.184:2375\n",
    "  m ID:\n",
    "  m Status: Pending\n",
    "  m Containers: 0\n",
    "  m Reserved CPUs: 0 / 0\n",
    "  m Reserved Memory: 0 B / 0 B\n",
    "  m Labels:\n",
    "  m Error: Cannot connect to the Docker daemon at tcp://10.136.3.184:2375. Is the docker daemon running?\n",
    "  m UpdatedAt: 2017-07-22T03:34:09Z\n",
    "  m ServerVersion:\n",
    "Plugins:\n",
    " Volume:\n",
    " Network:\n",
    "Swarm:\n",
    " NodeID:\n",
    " Is Manager: false\n",
    " Node Address:\n",
    "Kernel Version: 3.10.0-514.26.2.el7.x86_64\n",
    "Operating System: linux\n",
    "Architecture: amd64\n",
    "CPUs: 0\n",
    "Total Memory: 0B\n",
    "Name: kubernetes-master\n",
    "Docker Root Dir:\n",
    "Debug Mode (client): false\n",
    "Debug Mode (server): false\n",
    "Experimental: false\n",
    "Live Restore Enabled: false\n",
    "\n",
    "WARNING: No kernel memory limit support\n",
    "================================================\n",
    "\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
